Musical compositions communication system, architecture and methodology

ABSTRACT

A musical workstation system produces a display presentation in one of a musical composition responsive to musical composition data and responsive to one or both of input variables and a selected operating mode. The system is comprised of (1) means to provide the musical composition data (such as local storage (ROM, RAM, CD-ROM, hard disk etc.), or via a communications interface to an external device (such as another music workstation, a master controller, a computer), a memory, a selection subsystem, a controller, and a display subsystem. The memory selectively stores the received original musical compositions. The selection subsystem determines a selected operating mode and display format. The controller, responsive to the selection subsystem, provides means for selectively controlling the storing of the musical composition data in memory and selectively processing (e.g. altering) the stored musical composition data responsive to the selected operating mode and the input variables to produce a particular one of a plurality of processed results, such as external communications, operating, mode, transformation to derivative musical compositions, etc. The music workstation can communicate with one or more external devices, such as other music workstations, a master workstation, a controller, etc. The display system provides for selection of original compositions, creation of derivative compositions, distribution of compositions, monitoring of each performer&#39;s performance, group virtual performances, and also allows for local and distributed retrieval and editing of musical compositions, such as changing keys, pitch, tempo, and many other parameters. The music transformation can be performed locally or at the central or distributed music database, which stores the scores of musical composition. The musical composition data can be transposed via a controller, and can be transmitted to a plurality of the individual music workstations that then display the music composition, to permit playing by musical instruments or vocalists, and for location presentation.

This is a continuation-in-part application, under 37 C.F.R. §1.53, ofpending prior application Ser. No. 8/677,469 filed on Jul. 10, 1996, forMulti-Dimensional Transformation Systems and Display CommunicationsArchitecture for Musical Compostions, now U.S. Pat. No. 5,728,960.

I. FIELD OF THE INVENTION

The present invention relates to the field of music. More particularly,the present invention relates to a display system for displaying musicalcompositions, either batch or in a real time environment, and processingand communicating user performances.

II. DESCRIPTION OF THE RELATED ART

Music is usually only available in the written form for one (or a fixedset of) performer/instrument types in a fixed key. Adaptations orvariations of musical arrangements are complex and costly. Remotelylocated musicians are unable to effectively practice together. Smallcommunities each with only a few musicians are limited to practicingwith the few musicians they have.

Performers of music have many inconveniences to deal with. One suchinconvenience deals with the composing, distribution, and utilization ofmusic display presentation, traditionally sheet music. Another majorproblem relates to the inconvenience of scheduling and physicalgathering of multiple musicians (including instrumentalists andvocalists), which when combined in their performance provide a musicalensemble or orchestra. For example, high school band practice requiresthat all students be available to practice at the same time at the sameplace (i.e., the school music room). However, this creates difficultiesin that many students have other activities which conflict with bandpractice which is then incomplete. Additionally, when composing,musicians often will come up with an idea when physically not withanother musician.

Musicians typically work from sheet music. When composing, they writethe notes down on paper that has a number of staffs. If the musiciantransposes a composition from one key to another, the notes are alsowritten down on the staff paper. The scores for different instrumentsmust also be generated and written down. All of the scores are thencopied for distribution to other musicians and/or music stores.

When performing, the sheet music must be found, for all parts to beplayed, manually distributed, manually set-up, manually handled (turnpages, etc.). There is also an unfulfilled need for quick access to amore comprehensive database of music for the performing musician,whether he is solo or part of an orchestra. Also, musicians oftenperform audience requests, and require access to sheet music forrequested songs. Presently, there are various combinations of songscompiled in "FAKE" Books, usually by category (e.g., rock, country,blues, big band, etc.). This is only of limited help. Furthermore, theuse of paper sheet music is cumbersome and inconvenient; pages often getdamaged or lost, and indexed access is poor and slow.

This method of composing and distributing music is inadequate when themusic is used by a band or orchestra that requires hundreds of copies.If the conductor desires the piece to be played in a different key orcertain sections of the music edited to suit the conductor's tastes, thecomposition must be rewritten and the new transposed copy distributed tothe band or orchestra. This is a very costly, time-consuming, andlaborious task if the orchestra has a large number of members.

Additionally, if the composition does not have a part for a certaininstrument, the conductor must generate the required part from theoriginal composition. After the score for the required instruments hasbeen generated, the parts must be copied and distributed to theindividual musicians. This, again, is a very costly and laborious taskif the band has a large number of musicians requiring different parts.There is a need, therefore, for a more efficient way of transposing,editing, and distributing music scores.

Over the past many years, great advances have been made in theelectronic input, storage, and display of music. Electronic bands andorchestras are constructed using computers and MIDI equipment. Programsexist for personal computers (e.g., Apple Macintosh, DOS, and Windowsmachines) for an individual to use the computer for transposing music,composing music. Programs also exists for automatically inputting musicfrom direct performance (such as directly from a keyboard,electronically through MIDI converters (such as for string instruments),via pickups and microphones, and sequencers, tone generators, etc.) Togenerate digital data and/or music notation.

Musicians often perform both pre-planned and ad hoc compositions duringthe course of a performance. It would therefore be desirable to have theability to access a large database of musical compositions on demand. Itwould also be desirable to permit communication and synchronization of amusic presentation to multiple performing musicians who are playingtogether. It would also be desirable for a performing musician to havehis or her performance of the music input onto an individual musicworkstation, and stored, and analyzed by an automated system, and/orcommunicated to one or more other networked (remote) individual musicworkstations.

SUMMARY OF THE INVENTION

The present invention encompasses a musical presentation system. In oneembodiment, the system enables one or more users to select a musicalcomposition, and perform the selected musical composition at each of aplurality of individual music stands, independently capturing theperformance of a respective user and communicating the individualsperformance data to a master workstation (which can be standalone or oneof the networked individual workstations), which combines the pluralityof individual workstation performance data into a composited combinedperformance data and communicates said combined performance data back toall of the individual workstations wherein the individual workstationsprovide for audio (and/or vide and/or audiovisual) output representativeof the combined performance data (which represents the musicalperformance inputs for all of the communicating plurality of individualworkstations). The time between the users performing the segment ofmusic and that same user hearing the audio presentation of the combineddata is less than the time interval detectable by a human being. In apreferred embodiment, the plurality of individual workstationscollectively provide for synchronized display presentation of a selectedmusic composition's presentation, and provide for output of individualperformance data representative of the musical performance of the usercorresponding to the display presentation. Timing-synchronization isalso provided for to permit synchronization of the display presentationof the plurality of individual workstations, and for synchronization bythe master workstation of the plurality of individual performance datato construct the combined performance data. In one embodiment, themaster workstation generates a synchronization signal that iscommunicated to all the individual workstations from the masterworkstation. Other synchronization structures are also equallyacceptable, such as embedding timing synchronization data within theindividual performance data.

In another embodiment of the present invention, a musical presentationsystem enables a user to select from one or a variety of musicalcompositions, control the adaptation of the selected composition, anddistribute the edited version efficiently in a paperless environment.The system and process of the present invention also provides means forreceiving music from a number of sources. The user selects the desiredmusical composition from the source. The desired composition isdisplayed and/or stored in the system's memory for further processing bythe system prior to display and/or distribution.

In accordance with one aspect of the present invention, each of themusic workstations in an intelligent music composition communicationarchitecture provides for musical information, to be distributed for avideo or visual presentation of music in a user-friendly notation,and/or provides an audio presentation that can be selectively trackedand synched to the video presentation and/or tracks and synchs thedisplayed video presentation to a live performance, etc.

In still another embodiment, the system includes a user input deviceenabling selection of the musical composition, and optionally,permitting any user specified editing desired in the composition, and,in a preferred embodiment, permitting user selection of parameters (suchas the musical key in which the composition is to be played). The usercan then instruct the system to transmit the newly generated musicscores to one or more display subsystems (such as CRT's, LED's, LCD's,etc.), or to other systems. In the preferred embodiment, these displaystake the form of liquid crystal displays built into music stand basedsystems, also referred to herein as display stations or workstations.

This invention also relates to a musical presentation and/orcommunication system, and more particularly, to a system which permitsmusicians to view the score or other audiovisual or visual-onlypresentations of a musical composition, to permit the musician/user toperform the musical composition on the instrument of choice, as selectedby the user, and in the key of choice, as selected by the user.

In accordance with one aspect of the present invention, one or moremusic workstations are provided, consisting of a video presentationmeans, such as a display screen (CRT, LCD, LED, Heads Up Display (HUD)etc.) in conjunction with a computer-based system which in oneembodiment stores a database of songs and music which can be utilized bythe musician/user of the system.

In accordance with yet another embodiment of the present invention,there are provided numerous music workstations, such that different orsimilar instruments can each select a respective portion of a song to beperformed, such that all musicians performing the same musical piece areprovided musical presentation or representation in the same key topermit the playing of the music together. In a preferred embodiment, thesystem is capable of locking into a particular key (and/or instrumenttype) responsive to input variables (such as can be provided via aninput device such as a microphone or keyboard, or voice recognition, orcamera) to determine the desired key to be played in. In a preferredembodiment, the user can select an auto-transpose mode where all musicis thereafter automatically transposed. The transposition can take placeat a master workstation and then be communicated to the individualworkstations. In one of the illustrated embodiments, the transpositionof the music takes place locally at the music workstation which provideslocal intelligence. An original musical composition, such as selectedfrom a stored database of a library of music is then transposed inaccordance with transposition rules. Many options are available, andvarious transposition rules and methods are well known and documentedand in use on numerous products, such as keyboards which have built-incapabilities for transposition of music, as well as computer softwarewhich provides for transposition and notation of music.

In accordance with an alternate embodiment, the user connects to aremote database via wireless or wired communication to permitdownloading to the local computer of those musical pieces which the userhas selected for performance. The user music terminal, or the centralcomputer, can receive the transposed (derivative composition) version orcan receive an unmodified (original musical composition) version, whichit can then display and/or convert and transpose music as necessary foreach instrument and/or key.

In another alternate embodiment, a user can prearrange for downloadingof selected compositions via a remote connection service, where themusic user terminal can include a non-volatile storage memory permittingthe storage and replay of all requested compositions. Alternatively, acentral server can be provided, where multiple music end terminals sharea single central controller computer, or each have their own computerswhich share in a central server computer or with each other in adistributed architecture.

Alternatively, a non-volatile storage structure, such as a CD-ROM, cancontain a database of music which can be selected from for transpositionin accordance with the present invention.

The present invention provides a music communication architecture andmethodology that permits the synchronizing of music displaypresentations for multiple display stations performing the same musicalcomposition. Bands and orchestras can be constructed using multipleindependent music workstation display whether at one site or remotelydistributed. In one embodiment, the music display presentations providesone or more of the display of the musical composition, performance bythe user, and the output presentation for the combination of theindividual performance data for a plurality of individual workstations.In one embodiment, a master workstation is responsive to the individualperformance data from the plurality of individual performance data andprovides means for synchronizing and compositing the individualperformance data from the plurality of the individual workstations, andproviding a presentation output (audio and/or video) comprising thecombined virtual performance.

It is a further object of the present invention to permit the comparisonof a performer's performance parameters, such as parameter signalsobtained via a microphone and/or camera, of the performing artist'sperformance or execution as compared to the stored and displayed music.The comparison includes the pitch, timing, volume, and tempo etc. of themusic (such as through audio recognition) and critique the artist'sphysical movements (e.g., proper finger position, etc.) through visualrecognition. In a preferred embodiment, the music workstation systemprovides the performer and/or a conductor with a presentationperformance feedback indicating the quality of the performance ascompared to the stored or displayed music, such as any errors, wherethey occurred, etc.

It is a further object of the present invention to provide a systemwhereby any displayed music can be transposed (e.g., to a different key,or converted to include different or additional different instrumentsand voices than that in which the sheet music is originally displayed).

It is a further object of the present invention to provide automatedmodes of intelligent operation of the music workstations, and to provideresponsiveness to multiple forms of user input.

These and other aspects and attributes of the present invention will bediscussed with reference to the following drawings and accompanyingspecification.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B show a music presentation system in accordance with thepresent invention;

FIGS. 2A-2G show flow charts of the processes in accordance with thepresent invention;

FIG. 3 shows one embodiment of the display for the music displayworkstations and input devices in accordance with the present invention;

FIG. 4 shows a shared music database and stand alone workstationembodiment in accordance with the present invention;

FIG. 5 shows a music communication system in accordance with the presentinvention;

FIG. 6 shows a master workstation and slave workstations in accordancewith the present invention;

FIG. 7 shows an alternate embodiment of the present invention using oneor more of the workstations coupled to a master controller and musicdatabase;

FIG. 8 shows a marching band environment in accordance with the presentinvention;

FIG. 9 shows a person outfitted with a sensor body suit in accordancewith one aspect of the present invention;

FIG. 10 shows a movement and pattern recognition system in accordancewith one aspect of the present invention;

FIG. 11A illustrates a flow chart for an alternative embodiment of theoperation of the automated mode "A Mode" 240 of FIG. 2A, alternative tothat illustrated in FIG. 2C;

FIG. 11B illustrates the process flow for an alternative overalloperation relative to FIG. 2A of a music composition communicationworkstation in accordance with the present invention, wherein steps200-250 of FIG. 2A are redefined to show an overall operation which alsoencompasses a specific networked mode A Mode 5, as further illustratedin FIG. 12;

FIG. 12 illustrates the process flow for the networked virtualperformance mode 1150 of FIG. 11B;

FIG. 13 illustrates the process flow for automated mode 6 correspondingto process step 1153 of FIG. 11A, directed to synchronizing the displayto the performance, to resynchronize the display of the music to accountfor performer error;

FIG. 14 illustrates a communications architecture in accordance with thepresent invention, where each of the individual music stands 1410receives a music input 1411 and provides performance data output 1412coupled to and communicating with a master workstation which generatesan output of virtual performance data 1451 responsive to compositing theplurality of individual performance data signals which virtualperformance data 1451 is communicated back to all of the plurality ofindividual workstations 1410 which provide audio output responsivethereto;

FIG. 15 illustrates one example of data flow communication architecturefor a plurality of physically separate remotely located locations, suchas a teacher/master conductor/server location and each of a plurality oflocations that provide communication of performance data for users ofthe individual workstations at each of those locations, wherein timinginformation is utilized by the master workstation to reconstruct andcombine all of the individual performance data into a combined virtualperformance data output coupled back to the plurality of individualsworkstations which responsive provide audio output;

FIG. 16 illustrates a data word structure compatible with one embodimentof the present invention;

FIG. 17 illustrates a block diagram for a workstation (generally shownin FIGS. 5-7) and further comprising a network interface and a MIDIperformance data processor,

FIG. 18 illustrates a timing diagram showing the synchronization timingof the architecture as illustrated in FIGS. 14 and 15, and datastructure illustrated in FIG. 16, and showing the compositedsynchronized virtual performance data and the multiple individualperformance data; and

FIG. 19 illustrates one example of a memory work space structure for theindividual and master workstations in the virtual performance mode, inaccordance with the present invention.

DETAILED DESCRIPTION OF THE DRAWINGS

While this invention is susceptible of embodiment in many differentforms, there is shown in the drawing, and will be described herein indetail, specific embodiments thereof with the understanding that thepresent disclosure is to be considered as an exemplification of theprinciples of the invention and is not intended to limit the inventionto the specific embodiments illustrated.

In accordance with the teachings of the present invention, a system andmethodology are provided for music presentation and communication.Musical compositions can be input to the present invention from any oneor more of multiple sources, such as from prestored score images, livemicrophone, direct input from musical instruments or vocal directperformances, scanning in of existing printed score images (opticallycharacter recognized), cameras, visuals, etc. These inputs by the systemare used in the selective storage, composition, communication, andpresentation of the musical system of the present invention. The systemcan generate additional material automatically, or permit a user tomodify, communicate, display and/or reproduce the musical compositions.

Modification can be performed on rhythm, primary key, individual notes,chords, etc. The vast store of musical information stored in digitalnotation format and/or any video format, can be broadcast (analog ordigital) to a local music workstation or a master controller, which canalso be a local workstation. The master controller can be a stand aloneunit, or act as a server as well as its own stand alone unit, or simplyas a server to a plurality of other stand alone units. However, in theminimal configuration, only a single musical user station is needed.

In one preferred embodiment, the workstation is provided as a musicstand where the display presentation is a liquid crystal display (LCD).The LCD that can provide monochrome, gray scale or high quality colordisplays, depending on design and cost constraints and desires. Otherdisplay types can also be used. A touch-screen input can provide forsimplified and adaptive user input selections. An optional built-inmetronome function can also be provided for display presentation audioand/or video. A subsystem can also be optionally provided to permit themusic to be audibly reproduced at the workstation through a speaker orheadphone jack or other output.

It is well known in the art to convert user analog audio input into adigital format, ranging from straight Analog to Digital (e.g., A/D)conversion to processed data conversion to encoded digital music data,such as MIDI. Examples of MIDI include guitar input or other stringedinstrument input through microphones or directly to MIDI-converters, orvoice/non-pickup instruments through microphone converted to MIDI-input,or keyboard MIDI-input. Such input systems are commercially availablefrom numerous companies for numerous types of interfaces at numerousinterface levels. Similarly, numerous A/D converter subsystems arecommercially available at chip and board solution levels (such as fromAnalog Devices Corporation and from Mattrox Systems).

In accordance with one aspect of the present invention, themulti-dimensional music transformation system of the present inventionalso enables a user to select one or more musical compositions from alarger database from a plurality of musical compositions. The databasecan be stored locally within the workstation, on site, or remotelystored and transmitted to the user (such as over cable, wire, telephonelines, wireless (such as radio frequencies)). The user can alsooptionally edit the selected score of the composition (such as changingthe key and/or any note and/or timing, etc.) to suit his or her taste.The score (altered (the derivative composition) or not (the originalcomposition)) can then be transmitted to one or more displays such asliquid crystal or CRTs in the music stands of the band or orchestra. Thepresent invention, therefore, provides an efficient, paperless solutionto communicating, presenting (displaying), and optionally one or more oftransposing, editing, inputting, comparative testing-teaching,conducting, and disseminating music to one display or a large number ofdisplays. Each display can have the same, or a different, unique,customized presentation of music notation as appropriate per selection,responsive to a set-up by a system, automatically per predefinedparameters, and/or to user input. The score can also be printed out if ahard copy is desired.

As illustrated in FIG. 1, the music is stored, such as on a large harddrive or CD ROM jukebox, in a digital format as a music library (120).The music library (120) is coupled to a processor subsystem (115).Coupling can be wireless or cabled such as through a shielded cable,fiber optic conductor, switched connection (such as via phone lines),local, or remote. The processor (115) has the local storage capacity(e.g., semiconductor memory, disk storage, etc.) to hold the digitizedversion of the music composition transmitted to it on request from thelibrary (120). The music library can be local or proximately remote fromthe rest of the system.

In a wireless embodiment, the music library (120) is coupled to acommunications subsystem (such as a radio frequency transmitter) (125)that transmits the contents of requested compositions from the remotemusic library (120) to the processor (115) through the antenna (104) ofthe transmitter. The antenna (102) at the receiver picks up thetransmitted signal and the receiver conveys it to the processor (115).This embodiment enables the music library (120) to be remote and locatedat a great distance from the requesting site. The communicationssubsystem (125) can be a transceiver for bidirectional wirelesscommunication, or a transmitter for one-way wireless communication (suchas where the requests are otherwise communicated to the music librarysubsystem (120), such as via a wired connection).

As illustrated in FIG. 1A, a system controller, in the form of a musicstand (105C) with a liquid crystal display, is used by an operator(e.g., performer, conductor, etc.) to select one or more musicalcompositions. FIG. 1A illustrates two types of music workstationsstands. The workstation stand (105C) provides certain optional featuresfor a more full-featured stand, including as illustrated, speakers (140)both wireless and wired communications capability, and as illustrated,shows the processor with memory (115) as an external separate component.The music stand (105P) shows the integration of the processor and memoryinto the music stand itself, and also shows both wireless (antenna(101)) and wired connection (port (107)) to permit networkcommunication. Alternatively, the conductor stand (105C) could have allor part of the features integrated into the music stand (105C).Depending on the function for which the music workstation stand will beused, some or all of the features can be provided for that stand tominimize costs or optimize versatility. For example, in one situation,only the teacher or conductor needs the full-featured, full-poweredmusic workstation. In that case, the performers or students do not havea full-feature workstation, but rather a scaled-down version of theworkstation stand. In the preferred embodiment, a user input device(110) (such as a touch screen, microphone, keyboard, switches, voicerecognition system, visual recognition system, etc.) is coupled to theprocessor in a wired (such as over a cable or fiber optic link) orwireless (such as over an RF link or infrared link) manner forworkstation stand (105C), or directly to the processor, where it isbuilt into the system controller as workstation (105P). The user canselect an original musical composition from the touch screen of theliquid crystal display (135). The processor responds by storing thatcomposition in the memory (115) of the local workstation of the user asrequested.

Using the touch sensitive LCD (135), the user can now create aderivative musical composition. The touch sensitive LCD allows the userto enter the musical key in which the original composition will beplayed, edit any notes desired, and select the instruments and partsthat will be playing the composition. The composition as originallycomposed, and the derivative or modified composition can be played backto the user over speakers (140) so that he or she may listen (e.g., suchas to observe how the changes will sound) while optionally permittingsimultaneous viewing of the score on the presentation visual display.Once the score has been designated (e.g., selected, edited, etc.) to theusers (e.g., conductor's) taste, the appropriate portions (e.g., bymusical instrument) of the scores can then be transmitted for (optionalstorage and) display to the respective associated individual musicworkstation stands of the band members.

In a preferred embodiment each stand has an input device (110) thatpermits the user of the stand to select which instrument will be usingthe stand. (As discussed above, this input device can take the form of atouch sensitive screen or a number of buttons or switches or voice oraudio recognition, etc.)

In the preferred embodiment, each individual music workstation stand(105) can be directly and/or remotely programmed to addressably receive(and optionally to locally convert) and display the music score that isintended for the respective instrument type (user type) that will beusing (is associated with) the stand. As an example, the user of thestand (or a conductor) can input their selection of saxophone into theuser input device (110) of the workstation stand (105C), to program thatworkstation stand (105C) only to receive the musical score for thesaxophone (see FIG. 3). Then, the musical scores for all selected partscan be independently broadcast to all connected workstation stands, witheach individual workstation stand individually distinguishing andaccepting only its part. Alternatively, each workstation stand can beindividually addressed for separate broadcast reception of its ownrespective selected part. Additionally, the user of the stand canprogram the user input to select a musical part of a selected musicalcomposition (e.g., saxophone first chair) and receive only the musicalscore intended for that chair. This same procedure can be followed forother instruments within the band or orchestra. Alternatively, a singlemusic composition can be broadcast to all workstations, where eachworkstation has local intelligence (processing and storage) to permitlocal conversion for display at each workstation for the selectedinstrument for each workstation.

For wireless communications, the individual music workstation stands(105) are comprised of receivers (or transceivers where bidirectionalcommunication is desired) and antennas (101, 103) for receiving (ortransceiving) the radio frequency information from (and to) the masterworkstation (such as for the conductor). The music stand also has adisplay (such as an LCD (135)) for displaying the musical score intendedfor that stand.

Referring to FIG. 1B, the music workstation stands can either beidentical or broken down into conductor stands and performer stands. Aconductor stand (105CON) may have more functions and control than aperformer stand (105PER). A performer stand (105PER) might only have theability to receive and display musical scores, whereas the conductorstand (105CON) has the ability to select the musical score, change thekey of the musical composition, and perform other tasks only a conductorwould be permitted or required to do.

In one embodiment, an RF antenna for the stand (105) can be built intothe stand itself Alternatively, instead of using RF, the performer'sstand can be linked to the main (e.g., conductor's) stand usinginfrared, fiber optic cable, shielded cable, or other data transmissiontechnologies. As discussed above, the communications link can bebidirectional, such as to facilitate requests and responses tofacilitate the feedback of performance parameters or such that anyworkstation can be a master or slave, or used in combinations.

FIG. 2A illustrates the overall operation of the music compositioncommunication workstation. It begins by starting up the system (200).The system then provides a menu (201) that allows the user to select alisting of available music compositions. The user then selects one ormore compositions (210). If the user selects one from the menu that islocally stored, it directly retrieves the information. Alternatively, ifit's not something locally stored, the system couples (e.g. will dial upor go through a database or network) to a remote storage site andrequests and receives the selected compositions.

Any changes that are desired to the composition can be selected at thenext logic block (215). If there are changes (such as to the key, ornote editing, or selection of form of display or instruments), thenthose can be accomplished as illustrated at blocks (255) to (285).

If no changes are desired, the musical score for the composition that isselected is broadcast, transmitted, or otherwise transferred to theworkstation music stand (220). It is internally stored in the localworkstation music stand. Next, the score is displayed (225) on theworkstation display (e.g., LCD or CRT) or a video projection system. Thedisplay can also be part of an integral stand-alone workstation or aninterconnected group of components including a personal computer (suchas Macintosh, or DOS or Windows PC).

The display mode selection is then made (230). This permits selection ofan operational display mode, not simply choosing the resolution orcolor. The two main choices in the preferred embodiment are a manualmode (250) and an automated mode (240). In the automated mode selection(240), there are many sub-modes or options, such as the operational modethat permits the performer or user to do their performing without havingto tend to the selection of the portion of the music to be displayed orthe turning of pages. In the auto performance mode as shown on LCD(135P), there is provided the simultaneous displaying of the selectedmusical composition, and a display representative of the audioperformance of the user, and a divergence signal or divergence datarepresentative of analyzing the performance, preferably in approximatelyreal-time.

FIG. 2B illustrates the manual mode (250), which provides for usermanual selection of functions (252). There are many functions that theuser can select, even in the manual mode, such as hitting a button or atouch screen to cause the turning of the page of the display. Anotherfunction is to go back a page or to scroll forwards or backwards. Forthose who are vision impaired, another function can increase the fontsize of the music presentation.

Thus, there are many manually selected functions that can be provided.While the manual mode can have automated functions selected, it isdistinguished from the automated mode where control is partiallypre-defined without user assistance. In the manual mode (250), the userselects any and all features that are going to be provided (some ofwhich can be automated). The selected function is then processed (256).

Next, any ongoing needs are processed (257). These needs can include anyoverlapping automated function (not otherwise inconsistent with anyother selected function).

Referring to FIG. 2C, the operation of the automated mode "A Mode" (240)is illustrated. First, the user selection of the desired automatic modeis detected and responded to, illustrated as the auto-advance mode(242), the training mode (244), the performance mode (246), or any oneof a number of other modes (248) as is described in further detailhereinafter. For example, auto repeat mode can be selected bydesignating the start and stop points, and the number of times to repeata "looped" portion (or portions) of the displayed musical composition.Marching band mode (auto-advance based on metronome function, conductorcontrol, etc), auto-compose mode, and many others can also beimplemented. The order of selection of auto-advance, training, orperformance mode is arbitrary, and the user can alternatively decidefrom a menu where all are simultaneously presented as choices.

The display can advance the music by page option, or by a user selectionof one of many options (e.g., scrolling, tablature, video graphictutorial display, etc.).

Referring to FIG. 2D, the automated mode 1 for auto-advance operation(242) of FIG. 2C is illustrated, where the user has selected anauto-advance performance mode. In this mode "A Mode 1" (271), the systemtracks the performance by the user of the composition to the score(272). Performance refers to the actual performance by an individualperson (or people) who is (are) reading the musical score upon which theperformance is based. Whether that score is in tablature format, staffand clef and note notation, or some other format, the system generatesappropriate signals to permit comparison of the user's performance tothe musical score.

Based on a comparison, a decision is made pursuant to selection criteriaprogrammed into the system (such as the rate at which the piece is beingplayed, the time signature, the tempo, the rhythm, and the advancementof the music on the available display), the display presentation isadvanced (274 and 278). In some cases, the music might move backwards,such as with D.S. Coda. The presentation of the display tracks theperformance to permit smooth, uninterrupted playing or singing. Thecapability can be provided for the user to over-ride this auto-advance,such as for practicing where it is desired to keep going back oversections. In this case, a user over-ride option (276) is permitted toalter the automated operation. Upon cessation of user over-ride, thesystem can be programmed to stop, to automatically return to the regularauto-advance mode, or to process other auto-modes (270) of FIG. 2C.

Referring to FIG. 2E, the automated mode "A Mode 2" (244) operation ofFIG. 2C is illustrated corresponding to the training mode. In this mode,the system tracks the performance (280) of the individual user to thecomposition score, primarily for the purpose of permitting a criticalanalysis and comparison of the performance to the score (282). Thisanalysis determines divergence from the selected musical score, andreveals errors or deviations from desired performance goals (e.g. matchof timing of notes, duration of notes, pitch of notes, etc.), and todisplay those errors (284) (such as by audio or video means). Predefinedperformance goals provide the knowledge basis for expert system basedanalysis.

The system can then generate a graded score (286) indicating errors, andcan present it in numerous formats such as histograms, frequency oferrors, spacing of errors, etc. Identification of when the errors occur(e.g., only when going from slow to fast, or fast to slow), absoluteposition within the score and so forth, are also tracked and reported.Other expert system rules can be provided by music teachers which givethe necessary parameters for modeling expert system reasoning, as wellas guidance and suggestions on how to correct problems such as viadisplay text, graphics, audio, etc.

The comparison of the performance to the score in the training mode isfor the purpose of detecting the performer's compliance to parameters(such as the tempo, rhythm, filter, parameter, pitch, tonality, andother features that are adaptable or can be modified by performers).This parameter information is available and published in numerous forms.Thus, having provided this core set of parameters, the system canthereafter perform the training automated mode.

As illustrated in FIG. 2F, automated mode 3 "A Mode 3" is theperformance mode (246). In this mode, the operation is as in automatedmode 1 (auto-advance mode) except that no user over-ride is permitted.Its primary purpose is to accompany the performer during the entireperformance of a score as an automated page turner. The tracking of the"page turning" to the performance can optionally be based on inputs orcriteria independent of a performer's actual performance input (e.g.,microphone), such as a built-in metronome clock, a central control(e.g., a conductor or special user input), etc. Additionally,performance characteristics can be tracked, computed, and reported as inthe teaching and training mode. Training feedback can optionally beprovided real-time, or subsequent to completion of performance, toassist the performer as in the training mode. Alternatively, the scorecan be presented in a moving score mode (e.g., vertically, horizontally,or otherwise) or linear presentation as opposed to a real page turningdisplay.

FIG. 2G shows the operation of automated mode 4 ("A Mode 4") whichprovides for the processing of other automated functions selected by thesystem. These modes can include conductor mode, karaoki mode, etc.

In conductor mode, a conductor can control communications of signals tohis or her performer (such as "increase volume", or "increase tempo", or"play staccato"). Icons can be provided where the conductor simplytouches a touch screen (or other input mechanisms) to supplement hishand and body motions to permit more effective communication with theperformers. Alternatively, as illustrated in FIGS. 9 and 10, in a moreadvanced system version, the conductor's movements are first learned bya monitoring system, based on user definition and assignment of meaningsfor movement to provide an expert knowledge database.

This system provides for tracking of movement input such as in FIG. 10via video camera (1005) input of the conductor (1015) against a backdrop(e.g., blue screen) (1010) is processed by video processing unit (1020),or, as shown in FIG. 9, via body glove technology (gloves (935) orsensors (944) or sensor clothing (940) or head or eye movement trackingsensor (930) (such as used in virtual reality, flight simulation,avionics equipments (such as jets and space travel), and sports playersfor analyzing movement) to provide the necessary movement input. Thismovement input is analyzed utilizing the expert knowledge database toautomatically generate a display (video and/or audio) to provide localvisual and/or audio reinforcement on the local display (such asoverlaying on a then unused portion of the music score display as apicture in a picture) to permit audio and video reinforcement of theconductor's body language. Thus, "a hush" body language signal that isdirected towards a particular section of the orchestra wouldautomatically be interpreted to cause the system to indicate, and onlyon that particular section's respective displays, a message (e.g., bigface with a finger in front of it making a hush sound with a "hush"sound simultaneously output from a speaker). The conductor mode providesmany benefits to performance and communication.

For all these automated modes (e.g., A Modes 1, 2, 3, 4), trainingfeedback can be provided real time or subsequent to performance ateither or both of the perforner's workstation and a second (e.g.,teacher's) workstation.

The advantages of electronic music composition, communication anddisplay are many. In addition to those discussed elsewhere herein, acapability exists for expert system based artificial intelligent typeassistance where the expert system assists in many of the functionsperformed in musical composition and performance. For example, in theAuto-Compose Mode, if the words need to be changed to match the meter,equivalent terms can be chosen from the many sources such as thesaurus,dictionaries, rhyming dictionaries, encyclopedias, etc., to assist aswell. Phrases from poetry, selected and indexed by content or topic canbe re-expressed to create new works. Drum and rhythm sectionaccompaniment can be expertly suggested, as well as harmonies, melodylines to accompany chords, chord progressions to accompany melodies,harmonies to accompany a melody, and suggested musical instrumentgroupings to support a particular sound, rhythm, style, tonal quality,etc.

The expert system can be built from commercially available technology,including component hardware systems with supporting software, as wellas commercially available software packages which operate oncommodity-type personal and business computers such as the Macintosh byApple Computer, Windows and DOS machines based on the X86 and Pentiumprocessor technology of Intel, technology based on the Power PC and68XXX processor by Motorola, DEC PDP-11 technology, Sun workstations,etc. Custom microcomputer or DSP based system architecture on a chip canalso be constructed, as well as ASICs, custom or semi-custom logic.

The system can be designed to take advantage of expert system designknowledge. A database of rules and facts are provided, and accumulatedover time by the system in a self-learn mode. The expert system itselfhas the necessary logic to probe the user, monitor the performance, andapply the rules to provide feedback and reports to the user of skilllevel, errors, automated performance display, etc., starting with a basedefined set of rules, instructions, and a knowledge database specific tomusic.

The form of the musical score communication can be easily shaped to fitneeds. One example is MIDI (Musical Instrument Digital Interfacestandard) which has advantages such as of bandwidth of storage used, iswidely available commercially, is standardized, etc. However, signalprocessing, text, icon-based, object based, and various other forms ofstorage, user interface, and processing can also be applied to morespecific applications of product.

FIG. 3 illustrates one embodiment of an LCD display used for inputcontrol and for displaying the information from the processor andmemory. In the preferred embodiment, this LCD is a touch sensitivescreen enabling the functions associated with each displayed button tochange, and also for the displayed buttons to be moved around thescreen, depending on the function to be activated. The musical score maybe edited by the conductor, such as by touching the individual noteafter which he is presented with a number of notes to replace thetouched note. The lower portion of the screen displays instruments fromwhich the conductor can select which instrument will be playing thecomposition. After a button on this screen has been touched, a number ofsub-screens may come up, each with their own individual touch sensitiveareas and functions to be activated by those areas. Alternatively, inaddition to or instead of the touch screen, the system can provide inputvia separate key switches, voice recognition, etc.

As an example, if the conductor touches the transmit key on the mainscreen, he will be presented with a screen showing all of theinstruments that he has selected for that piece and a button labeled"ALL". He may now transmit to each individual music stand or bydepressing the "ALL" area, transmit to the entire orchestra.

The music library can be contained ("stored") on non-volatile storageeither locally or at a remote central site containing the entire (or asubset) database of all possible music (that is then downloaded to localstorage on request, either real-time at performance time or in advance.)

Alternatively, the music library can be provided on storage medium thatcan be easily transported and used on site locally with the presentationsystem. Thus, for example, disk drives, cartridges, FLASH RAM cards,plug-in memory modules, or a CD-ROM or multiple CD-ROMs in a CD-ROMchanger can be used to store and contain massive data libraries onmusical compositions. While this would be a more expensive route thanshared use of a central library, requiring each musical group to obtainlibraries on all possible compositions they may want, it has theadvantage of speed, flexibility, no need for communication with aseparate remote source, and creates a whole new mass marketing area(such as for CDs or Digital Audio Tape (DATs)). Another way of utilizingthis technology is to maintain a history of music used, either with theremote music library or local music library. This could be done for manyreasons, including copyright royalty assessment, determining a historyof musical performances and requests for future use in determiningperformance itineraries, etc. Alternatively, a hybrid of locally storedand centrally shared libraries can be utilized to optimize cost, speedand flexibility benefits.

In accordance with another aspect of the present invention, each displayworkstation can also provide the ability to convert performed musicalcompositions into annotated musical compositions, generating theappropriate musical notation (e.g., staff, tablature, MIDI), notes, timesignature, key, instrument, or user type, etc.

The display workstation can be implemented as a totally self-containedworkstation, where each workstation contains its own processingsub-system, optional communications interface (such as wireless orcable) for network use, input/output interface including one or more ofa user input keypad, a speaker, a microphone, joysticks, push buttons,etc. Each of the stand alone workstations can then operate with a localdatabase or couple to a shared music database as illustrated in FIG. 4.

The stand alone workstation(s) (105), are coupled to the shared databaseinterface (405), and can either couple remotely (e.g., via phone lines)to the remote shared music database or to a local shared or dedicatedmusic database (410). The shared music database (410) can either beprimarily a storage means (e.g., hard disk or CD-ROM), or can include aprocessing sub-system (420) for local intelligence. In one embodiment,the stand alone music workstation includes the shared music database(410) and interface (405), non-volatile local storage medium for theshared databases (410), and a local processing subsystem (420), and canoperate completely stand-alone. In an alternate embodiment of thisstand-alone device, the shared database interface is contained in thestand-alone workstation (but not the shared music database or processingsubsystem), and provides capability for communication with a storeddatabase (410) remote from the stand-alone device.

In either of these embodiments, an alternate additional embodimentprovides capability for each stand-alone workstation to function as amaster stand-alone, or a master or slave workstation within aworkstation set including multiple stand-alone workstations, wherein oneis designated master and the rest are designated slaves. The slaveworkstations in this configuration receive communication of musiccompositions to be displayed from the master workstation, therebypermitting one shared music database to be communicated among allworkstations which are a part of the group. It is to be appreciated thatthe shared music database function can be distributed in many differentways among the workstations, or separable from and independent from theworkstations. The choice is simply one of design, and the illustrationherein should not be taken in a limiting manner.

In one embodiment the master workstation has complete control over theslave workstation. Anything displayed on the master workstation is alsodisplayed on the slave workstation. It is also possible for the user tomask certain portions of the display of the master workstation before itis displayed on the slave workstation. In this manner, the conductor,using the master workstation, can transmit to the slave workstationsonly that information that is required by the orchestra members.

In an alternate embodiment, the slave workstation communicatesperformance parameters or deviation signals to the master workstation,for error analysis feedback.

In accordance with another aspect of the present invention, means areprovided wherein a plurality of individual workstations are coupledtogether in the network configuration to provide for networkedcommunication of musical performance data wherein each of the individualmusic workstations provides for capturing the performance data for ausers performance and communicating that performance data to a master ora conductor workstation which synchronizes and combines the plurality ofindividual workstations performance data to create a composite virtualperformance data output which is recommunicated back to all of theindividual workstations in approximately real time, so that theindividual workstations can receive the composite virtual performancedata and provide an audio output (and/or visual presentation) of thecombined composite virtual performance data including all of theindividual workstations users performances. While the networking can beused in conjunction with other features and embodiments of the presentinvention, including communication of musical compositions for display,and other musical performance data analysis and data communication, aswell as inter-musician interpersonal communication, the networkedembodiment of the present invention permits synchronized virtualperformance thereby permitting multiple remotely located individualworkstations to physically separately perform with the benefit ofhearing in approximately real time the combined result of all theperformances at the individual workstations with their own individualperformance.

In accordance with another aspect of the present invention, means areprovided to permit a user of the music workstation to accomplish atransposition of a musical composition in pitch, tempo, and otherwise.In a preferred embodiment, the lead voice or instrument can audiblyindicate the key via the microphone input or via another type of inputstimulus. The workstation can analyze the user input, determine the key,pitch and tempo for a musical composition being partially performed bythe user, and adjust and transform the composition to be displayed inthe new user desired key, pitch, tempo, etc., either solely for use onthat workstation, or communication for use on one or more otherworkstations. In a networked version, this user input can also becommunicated to other workstations for use by one or more of theworkstations in transposing, or communicated to a master workstation,which transposes and rebroadcasts the transposed composition.

Alternatively, the user can input the pitch, tempo, and key via the userinput (e.g. keypad, joystick, push buttons, voice recognition, playingof an instrument, etc.) and the system performs the transformation anddisplays (and/or prints out and/or audibly performs) the modifiedtransformed composition for the user. Additionally, where a musicalcomposition is written for one instrument and a different or additionalinstrument version is desired for simultaneous performance, the user canindicate the other instruments via the user input, and the system willgenerate the appropriate displays. The workstation can also provide anaudio output of the transformed musical composition, either for theindividual additional instrument or voice transform and present it, orfor the composite of additional versions and the original version, tohear the blended piece.

Referring to FIG. 5, a music communication system is illustratedcomprising multiple workstations (500) each comprising a display (510),user input such as a keypad (522), a joystick (524), push buttons (525 &526), a microphone (527), and a speaker (528). The workstation alsoincludes communication interface means such as a wireless interfaceincluding an antenna (531), or alternatively or additionally a wired orcabled communication interface (540). Each workstation further includesa local microcomputer subsystem (550) that provides local intelligenceand management of functions in the workstation.

In the networked embodiment, where multiple physically separatelocations each having one or more individual workstations provide forcommunication of performance data from the individual workstations andpresentation by the individual workstations of the combined virtualperformance data at the individual workstation, to provide for a virtualperformance. In this case, the communications interface would utilizeslightly different structure, such as a phone modem (analog modem), acable modem, ISDN as between locations, etc.

Communications interfaces of various types are well known andcommercially available. At the present time, they are available forpurchase at the chip, board, or system level. In fact, many single chipmicrocomputers include communications interface capabilities, wired orwireless.

The workstation further includes an optional musical instrument input(562) and a musical instrument output (564) that permit the coupling ofa musical instrument via a musical instrument interface (570) directlyto the workstation. Thus, a keyboard, electric guitar throughappropriate input, or a microphone input through the interface (570)permits instruments or voices to be directly input to the workstationfor direct input independent of the microphone (527).

The instrument output permits coupling of the instrument input signal,either directly fed through or as modified by the workstation for outputto the appropriate public address or amplification and presentationsystem or separate analysis system. The workstations are coupled eithervia wired or wireless communication to a processor subsystem (580) thatincludes a processor, non-volatile memory, read/write memory and aninterface to a non-volatile storage medium (582).

The processor subsystem (580) includes an appropriate communicationsinterface, such as a communications interface (540) for wired interfaceor (532) for wireless interface including antenna (533). The processorsubsystem couples to a non-volatile storage medium (582) containing,among other things, application programs, transformation programs, andeither a shared music library interface application program or theactual shared music library and access program.

As described above, the processor subsystem (580) and non-volatilestorage (582) music library can be built directly into one of the musicworkstations (500) to be a master, with the other workstations beingslaves, that can either include the processor subsystem and non-volatilestorage or can be lower cost dummy slave terminals. As illustrated inFIG. 6, a first master workstation (300) provides a basic workstationsubsystem (200) plus contains the processor subsystem (280) andnon-volatile storage system (285) as a part thereof so as to provide acomplete stand alone music communication system, and be capable ofacting as a master or master/slave. This master workstation(s) (300) canfunction as a stand alone, or can couple to one or more otherworkstations, including one or more masters (300) and/or one or morenon-master workstations (105).

The multiple connected workstations can operate as stand aloneworkstations using their local intelligence for displaying downloaded orresident music compositions. They can also interact in a master/slavelinked environment, where one of the master workstations (300) asserts amaster status, and all other inter-connected workstations, whetherworkstations (105) or master/slave workstations (300) operate in a slavemode coupled to independent on the designated master. Additionally,masters can communicate between each other for a master/master networkconfiguration.

Alternatively, the multiple connected workstations can operate togetherin a networked virtual performance mode, or in a networkedcommunications mode. A dedicated stand alone or distributed masterworkstation architecture can provide for the coordination andcombination and synchronization of the multiple individual performancedata outputs into a combined virtual performance data output.

Referring to FIG. 7, an alternate embodiment of the present invention isprovided where one or more workstations (105) include, at a minimum, adisplay of the music notation. These workstations are coupled to amaster music communications controller (415) that provides for aseparate user input (411) which provides input interface, such as to aMIDI status stream, computer data links (such as RS232, modem data link)etc. that designate requested musical compositions, transformations, anddisplay requests for various ones of the coupled workstations.

In an alternative embodiment, the master music communications controller415 provides for additional functionality including virtual performancemode, wherein the input interface (such as the MIDI stream, computerdata links, etc.) provide one or more of musical compositions data fordisplay, transformation information, display requests, user individualperformance data, and wherein the workstations respond to the mastermusic communications controller to couple their individual performancedata and receive back the combined virtual performance data.

The workstations (105) access the music database storage means (420)that provides the data for the requested music composition via themaster controller (415). The master controller (415) displays both therequested music composition as well as user interface communication forthe music communication system to be displayed on either a dedicateddisplay (416) or on one of the workstations (105) as designated by themaster controller (415). The music database (420) can either be local,or can be via a data link (e.g., phone line, RF, otherwise). In oneembodiment, a motion sensor subsystem (422) monitors motion of a targetperson and responds in accordance with predefined movementinterpretation characteristics parameters, such as for a conductor.

In a preferred embodiment, the user input means (411) is comprised of akey switch device, such as a touch membrane keypad or capacitance touchsurface. Alternatively, in one preferred embodiment, the user input isprovided via a touch screen technology. Touch screen technology permitsthe display of user interactive icons and legends including text andgraphics making possible unlimited customization of user input structureaccording to task needs. Thus, specific switches or sequences of touchesto the touch screen can be associated with common use icons from thetask being performed in conjunction with words to provide ultimateclarity. User error is virtually eliminated, with the aid of automaticentry error detection, such as defined fields, mandatory fields, etc.

Alternatively, the microphone input (527) can provide for coupling ofuser speech to a processor sub-system (580) that uses any of a number ofcommercially available and well known speech recognition algorithms.These algorithms provide for speech recognition input control, eithersolely or as a supplement to touch screen or other tactile inputmechanisms.

In a deluxe embodiment, an output (421) is provided that permitscoupling of an external display, such as a color monitor, projectionunit, or other display presentation system including one or more ofaudio, visual, and audiovisual.

Additionally, the display presentation output of the workstation canprovide for an audio output presentation of the musical performance,either generated by the work station responsive to the music compositiondata, to the users performance, to the combined virtual performancedata, or responsive to an external source. Additionally, a visual oraudio visual presentation can be provided to provide informationfeedback to the user on both their individual performance as well asfrom and between individual workstations and/or the master controller orconductor workstation.

In accordance with another aspect of the present invention, means areprovided for moving through the printed (displayed) notation of themusic in synchronization with the live performance from the displayedmusical notation.

Musical notation is used, in the generic sense, to refer to any way ofconveying musical performance instructions including but not limited tocommon musical notation with staffs, notes, sharps, flats, and clefs,extending to written instructions in text form to supplement this orsupplant or partially replace, as well as alternate forms of expressionsuch as chord charts, words and chords (letters), tablature, any video,graphic, audio, audiovisual or other display presentation or combinationof the aforementioned types of presentations.

An annoyance in performing music using any written notation (whether onpaper or displayed on a presentation apparatus such as a screen) issmoothly performing music and timing "flips of pages" (or thecommunication of change to a third party, such as touching a key orbutton). This is especially true when the user is marching and bothhands are required simultaneously.

In accordance with one aspect of the present invention, means areprovided to accept inputs from one or more sources that initiates a"page turn". Types of inputs include conventional touch input apparatus(such as key switches or capacitive touch pads), motion sensing gear,and automatically when operating in the operational mode of Auto Mode.The motion sensing gear can be for a portion of the performer's body,such as a head tilt sensor or an optical eye movement sensor, etc.

Additional types of inputs that can initiate a "page turn" include voiceor sound recognition apparatus built into the microcontroller system.This apparatus has the ability to use pattern recognition specific tothe sound or user voice and words being said (for extremely highaccuracy). Of course, any type of user actuated device such as a foot orhand switch, or head motion device, or sound or voice recognitionsystem, in a preferred embodiment, is selectively permitted to controlthe override of the normal progression of the music's play.

The override may cause the progression to go backwards or forwards inthe music score irrespective of the normal reading of it. Theperformance mode AutoMode blocks the user override to permit performanceaccording to proper material timing and either progresses responsive tothe music composition data timing, or in an optional embodiment, to theperformer. This automatically moves through the musical score as writtenand preferably shows an indication of metronome time and an indicationof the proper place in the score where the performer should be for thatinstrument at any specific time. This is especially valuable in aconductor mode of networked communication, where a conductor couples toone or more music workstations.

The user's performance can be compared to the score, and feedback can beprovided to the performer as to the quality of their performance.

In the performance monitor mode, for a single user or multiple users,the user (or a remote teacher or conductor) can indicate the rate atwhich he feels the performer should be performing. A microphone input onthe music workstation samples the user's actual performance and permitsproviding a graphical mapping (for the user or teacher/conductor)showing the relative synchronization of the performer's actualperformance versus the conductor's desired performance.

In an alternate automatic advanced mode, the display of the musiccomposition is synchronized to the performers actual performance. Thus,rather than simply indicating visually for the teacher/conductor or userwhat their relative performance was to the written displayed musicalcomposition, the relative performer to written music synchronizationinformation can be utilized to adjust the display rate of the actualmusical composition to match that of the performer.

With use of appropriate sound baffling, a plurality of instruments cansimultaneously be monitored and controlled by the conductor, so long aseach instrument's output sound pattern is communicated directly to arespective workstation. The output of each of the workstations can thenbe coupled to the conductor's master workstation for further analysisand processing.

A workstation for an oboe may have a built in slide boom with afeatherweight microphone to be able to receive sound input from theoboe. Electric instruments, such as guitars, keyboards, and otherelectrical analog signal sources can be fed directly to a line inputthat is appropriately buffered and filtered. Signal input can also beaccommodated through a MDI-interface sub-system that permits bothutilization of data in workstation to workstation communications andutilization of MIDI-output at the station where the data was input.

For networked virtual performance, and for one aspect of output displaypresentation, the utilization of MIDI input and MIDI output, at each ofthe individual workstations and at the master controller workstation,permits the capture of the user performance and conversion to individualperformance data which includes time synchronization information whichcan be communicated to the master workstation, which synchronizes andcombines the individual performance data to generate combined virtualperformance data which is then communicated back to the individualworkstations which utilizing their MIDI output interfaces provide fordisplay presentation (e.g. audio output) of the combined virtualperformance data. Additionally, even where virtual performance mode isnot selected, the provision of MIDI interface input and output on theworkstations have other multiple beneficial users as discussed elsewhereherein. Additionally, other types of user performance to userperformance data input devices and transducers can be utilized, as arewell known commercially available including variations of analog digitalconverter input devices, audio signal capture, etc.

By combining the conductor and performance mode operations, theworkstation can be enhanced to provide training and comparison ofperformance to actual music.

Some music is only available in anotated forms where there is not anexisting signal showing proper synchronization of the signals. Thus, acontroller subsystem (such as (580)) provides for real time conversionand analysis of syntax of the music notation, in conjunction with aprecision clock metronome, and provides an indicator (such as color orother highlighting or bolding or accentuating) of the relative timing ofthe performance relative to a place in the sheet music (or other form ofmusical notation).

Existing forms of music notation can be converted manually, or can beconverted automatically by scanning in sheet music, recognizing (usingoptical character recognition) the various elements of the music, andfacets and specifics of the syntax in the form of notation including itsconstants and variables and protocols, and integrating via an artificialintelligence type expert system that notates, highlights, andaccentuates via synchronized metronoming of time signature to music. Anyof a variety of other means of converting music can also be used, suchas direct input of musical performance signals processed via softwarethat converts it into musical notation. Such software is commerciallyavailable, such as from ARS NOVA, Wildcat Canyon Software, Mark of theUnicorn, Inc., and Passport Designs, Inc.

Since the music notation is now in computer usable form, it is now aneasy task to communicate, display, compose, alter, and transpose music(such as in key, for types of instruments or voice parts, and harmonies)via well-known techniques.

Additionally, where the user input is converted into user performancedata for the workstation, the users individual performance data is alsonow in computer usable form and if appropriately converted fromperformance to performance data, includes appropriate synchronizationdata relative to master controller performance synchronization signal.Thus, both the music notation is in computer usable form (making it easyto display, alter and analyze/compare), and the users performance is incomputer usable form (digital individual performance data), it ispossible to provide intelligent operation and analysis and utilizationof both the music composition information and the user performanceinformation, to provide for various automated modes and features to theusers Implementation can also be in a custom design comprised of amicroprocessor, non-volatile storage memory, read/write memory, andwhatever additional peripheral circuitry is needed (such as areavailable in ASICs, or single chip micro-computer chip sets includingCPUs, DSPs, A/D, and other peripheral support circuitry). These singleor multiple chip solutions can be utilized to create a dedicated systemto perform complete music workstations performance criteria to supportan extremely low cost, high volume music workstation solution.

A new form of communication is created in that both the process ofcommunicating via standard notation is respected and adhered to, whileat the same time permitting interaction and communication of music mediasignals ranging from simple analog, digitized (analog to digitalconverted), individual performance data (representative of the user'sperformance), and can optionally include timing/synchronization data.

A multi CD ROM changer accommodates indexed storage of hundreds ofthousands to millions of musical compositions to permit complete standalone operation of the user music workstation. Alternatively, anoptional built-in or external modem can be provided to permitinter-communication with a remote central music database managementsystem that permits both communication and down loading (and disconnect)for stand alone operation. Thus the workstation can stay on-line,pulling up music as needed, or can request a single or multiple piecesof musical works be provided to it, that are then down-loaded from thecentral database manager. The user workstation then disconnects from themusic database management system, and thereafter operates stand alonewhere all desired music is stored locally in storage (preferablynon-volatile). Storage can be semiconductor, magnetic, optical or anyother medium.

The use of virtual reality technology, including motion sensors and bodygloves, permits monitoring of various other things (as shown in FIG. 9).For example, as shown in FIG. 10, a camera in conjunction with analysislogic, such as expert software, can monitor motion of role modelbehavior and compare performer behavior. Hand, finger, arm, leg, eye,head, body, and mouth movements can all be monitored and constructivecritical feedback can be accumulated, analyzed, and fed back to the useror teacher, for performer training, or performances, or for conductorcommunication.

The input of monitored movement data is provided to the userworkstation, permitting precise mechanics training such as fingerposition, the angle of striking of strings relative to the neck of aviolin or guitar, or they can be used to permit the virtual performanceof music by a performer using a virtual link apparatus such as a virtualreality glove and head movement detection apparatus. The user can thenperform a piece with their own personalization without any musicalinstrument in fact.

For example, the guitar portion for a piece of music could be displayedin notation form and actually performed according to the timing ofmovements of the user's fingers (either actual fret positions, or onlytiming information). To add further reality, a mock guitar, keyboard,flute, or other instrument can be used and combined with virtual effectsto provide for music performance and personalization. Thus, forentertainment purposes, users could perform as part of a symphonyorchestra playing a violin portion. If they performed out of time, theywould hear their instrument's performance out of synch with the rest ofthe orchestra's performance.

There are numerous ways to embody the conductor movement interpretationsystem. As illustrated in FIGS. 9 and 10, one is utilizing the bodymovement detection apparatus prevalent in virtual reality, sportsmedicine, etc., as discussed above, to identify specific movementpatterns or signal parameters associated with certain movement patterns,to initiate a display presentation, audio, video, or audiovisual toprovide a presentation associated with movement of the conductor.Alternatively, other techniques can be used such as taking the videofeed from a video camera or other video source (e.g. VCR) and having theconductor interpret his movements and assign them unique meanings, tocreate a lexicon of his movements and corresponding meaning.

For example, rapid downward movements of the hand from up to down, in acertain manner, indicate "decrease the volume." When he points at aparticular section at the same time as he is doing that, he isindicating that only that orchestra section is to reduce volume. In thismanner, either camera input of movements, glove sensing of movements, orother techniques (such as audio, ultrasonic, etc.) can be used to trackmovement to permit associated meanings to be attached or indexed toparticular signal parameters or parametric signals of the meaning of themovement parameters as provided by the conductor input device. Forexample, in the case of the virtual reality glove, that input would bethe signal output of the glove as interpreted by associated software ina processor (such as a PC or a MAC). Alternatively, for example, in thecase of video camera input, it could be pattern recognition or analogsignal comparison to determine the presence of certain signal patternsindicating to the system to initiate automatic communication of aconductor presentation. In so doing, the conductor is able to rapidlyconvey his meaning, focus it to a particular group of instruments, andbe done with it. He doesn't have to focus very long or concentrate tomake sure they've gotten his signal. Instead he can focus on listeningto see if they got his message.

FIG. 8 illustrates an alternate embodiment of the present invention. Inthis embodiment, the workstations are remote units (801-803) used by amember of a marching band. Each of the remote units (801-803) areequipped with receivers (810-812) that receive musical compositionstransmitted to them. Remote units controllers (820-822) control theoperation of the remote unit (801-803). The musical composition isdisplayed on the remote unit's displays (830-832) which displays can bean LCD multiple line display providing low cost, low power usage, andhigh visibility/readability, and with Auto Advance Mode, the displayautomatically scrolls as the music is to be performed.

Each remote unit (801-803) can be mounted on the instrument on or inplace of the lyre. The remote unit's antenna (840-842) can be separatefrom or built into the remote unit or the lyre.

A transportable main unit (850) is used to transmit musical compositionsto the remote units (801-803). The transportable main unit (850) iscomprised of a controller (806) for controlling the transportable mainunit (850), a music database storage medium (805) containing the datafor the musical compositions to be played by the band, and a transmitter(804) for transmitting the musical compositions to the remote units(801-803). This main unit can be in the form of a suitcase or briefcasesize item. The main unit can also be provided built into a van that isdriven around with the band or as a small self-contained portable unit.In accordance with this embodiment, the band can play a virtuallyunlimited number of musical compositions without the problem of carryingthe music with them in paper form. It also relieves the band members ofthe problems of changing music and changing pages while marching. Asdiscussed in the above embodiments, in the performance mode, the musicalscore is automatically scrolled across the screen display (830-832).Additionally, a keyboard and/or microphone can be attached to thetransportable main unit allowing the conductor to send messages to theremote units via displays (830-832) or via a speaker associated withunits (801-803). This allows the conductor to send instructions to theband (such as to take a certain route, or play at different volumes orspeeds). With bidirectional communications and user performancefeedback, the conductor can also monitor for errors.

FIG. 9 illustrates a conductor, stage hand, or other person with asensor glove on each hand (935) and a head and eye movement monitor(930). The figure also illustrates the conductor wearing full bodysensor equipment (940). Either embodiment or a combined embodiment canbe used to map body movements. If only the gloves (935) or body sensors(944) are used, the movement of the glove or sensors can be captured bya video system, as illustrated in FIG. 10.

Other methods that capture motion rely on specialized sensors (944)placed on a performer's joints, such as via a sensor body suit (940).Once motion has been filmed or analyzed, a data set is produced tointerpret that movement into Cartesian co-ordinates. These co-ordinatesprovide the spatial location of each of those markers. This informationis then cleaned up and input to an animation package.

FIG. 10 illustrates a video camera (1005) and a standing conductor(1015) (or performing musician to be tracked or virtually linked toperform), with or without a blue screen (1010) behind him. The videocamera (1005) feeds a video signal to the video processing system (1020)that utilizes signal processing to provide signal pattern recognitioncapability. The blue in the screen is filtered out in the signalprocessing such as by an Ultimatte process.

In one embodiment, the conductor is wearing a sensor equipped body suit(940) and gloves (935) of FIG. 9. In another embodiment, the conductoris wearing only the sensor equipped gloves (935) of FIG. 9. In stillanother embodiment, the conductor's movements are picked up by the videocamera (1005) and processed without a sensor suit.

Simple things, like looking for the conductor's rapid hand movements,focusing on specific hand movement areas, facial and head movement, armmovements, and body language can all be programmed into the recognitionknowledge base. Some of the technology for complete mapping of bodymovement that exists in making video games of today are illustrated inVideo Systems magazine, page 42, October 1995, Vol. 21, No. 11, and NEXTGeneration magazine, pages 49-54, October 1995, both incorporated hereinby reference.

In any event, having now obtained knowledge related to recognition ofthe movements, the system can interpret them and utilize them to conveypresentation information to the ensemble or orchestra or studio members,or to analyze a performer's movements, or to permit a virtualperformance. One example would be a large screen television or multiplelarge screen televisions for viewing by the members of the viewinggroup. Alternatively, each music stand could provide for a picture inpicture display of special movements of the conductor in areas of thedisplay where music is not currently being played. Since the stand canhave the intelligence to compare the performed music to the playedmusic, that embodiment permits display of the message in portions of themusic display area which have already been performed or are not going tobe performed for some time (e.g., at least ten seconds in eitherdirection; other criteria could alternatively be used, and can be set upfor desired characteristics of the performing environment).

Voice recognition and response to conductor commentary can supplementthe system. The system could record the message, interpret to whom theconductor directed the message and convey it audibly or translate itinto a text or icon display as a part of the system's audiovisualpresentation.

Referring to FIG. 11A, an alternative embodiment of the automated mode"A Mode" (240) of FIG. 2C is illustrated. The illustrated process forthe automated Mode 240 of FIG. 11A includes process steps for the AutoAdvance A Mode 1, Training A Mode 2, Performance Mode A Mode 3 which areequivalent to the illustrated A Modes 1, 2, and 3 of FIG. 2C. However,as shown in FIG. 11A, at step 1148, the decision is made whether theconductor automated mode is selected, and if a "yes" decision, thenautomated Mode 4 is entered at Step 1149, and if a "no" decision, thenlogic goes to the networked virtual performance mode A Mode 5, asdecided at decision logic step 1150, and responsive to the selection ofthe networked virtual performance mode, A Mode 5 (1151) is executed. Ifnetworked mode A Mode 5 is not selected, then processing proceeds todecision logic step 1152 to decide whether the synchronization of thedisplay to the performance (Mode A 6) has been selected, and if so, thenthe processing proceeds at step 1153 to provide A Mode 6 functions. Theautomated mode processing ends at step 1154. It is to be noted that theautomated modes can be cumulatively selected, so that the networkedvirtual performance can include any or all of the other automated modes,including A Modes 1, 2, 3, 4, 5, and 6, or other A Modes which may beprovided. It should also be noted that the tracking of the performanceby the user relative to the presentation of the musical composition(step 272 of FIG. 2D) provides the first part of the logic necessary forthe A Mode 6 (synchronization of display to performance), as illustratedin FIG. 13 in further detail.

Referring to FIG. 11B, there is illustrated an alternative overalloperation of a music composition communications system, alternative tothat illustrated in FIG. 2A, showing additional automated modesincluding the network virtual performance mode A 5. FIG. 11B illustratesan alternative control logic structure for the network selection, wherethe decision logic proceeds in an analogous manner to FIG. 2A. Theoperation starts at step 1100, and a mode is selected at step 1110. Acomposition is selected at 1120, and if the decision logic 1120determines the composition has been selected, then processing proceedsat step 1130, which provides for selection of the mode. If nocomposition has been selected, processing proceeds back to the selectcomposition step 1110. Once the composition has been selected, then aMode is selected as illustrated at step 1130, and the decision logic ofselect Mode 1140 determines whether the selected mode is that of (1) adisplay mode, in which case processing proceeds to step 220 of FIG. 2A,(2) a modified composition mode, in which case processing proceeds asfrom step 215 of FIG. 2A going to the yes branch, or (3) the select modedetermines that the networked virtual performance mode has beenselected, in which case processing proceeds to the networked modeautomated Mode 5, step 1150, as illustrated in further detail in FIG.12.

Referring to FIG. 12, there is illustrated the networked virtualperformance process flow, step 1150 of FIG. 11B, in greater detail. Atstep 1150, the automated mode of networked virtual performance has beenselected. This automated mode 5 is processed by proceeding to verify theselection of the composition. At step 1220, the user's performance iscaptured (e.g., by a microphone or via MDI) and converted to individualworkstation performance data. This data is then broadcast by each of theplurality of individual music stands to a designated master controller.At step 1230, the controller receives and combines the plurality user'sindividual performance data to generate a combined performance datarepresentative of the synchronized and combined plurality of individualperformance data This combined performance data is then broadcast to allof the individual music workstations. At step 1240, each of theplurality of individual music workstations provides an output (e.g.,audio) responsive to the combined performance data, which isrepresentative of the combination of the plurality of users audioperformances of the music composition, synchronized to the displaypresentation of the music composition. Timing synchronization data isprovided to synchronize the plurality of individual music workstationsindividual performance data to one another and to the selected musicalcomposition's display presentation, to facilitate synchronization forpurposes of combination and audio playback at the individualworkstations, in approximately real time, so as to provide apparent realtime feedback of the individual user's performance in combination withall other individual users, played back at each of the individualworkstations. At step 1250, analysis is done as to whether thecomposition is done. If so, processing completes at step 1260 and endsthis automated Mode 5. Otherwise, if processing is not done, that is themusical composition is not over, then processing proceeds back to step1220 as illustrated.

Referring to FIG. 13, there is illustrated automated Mode 6corresponding to synchronizing the actual display presentation to theuser's performance, providing resynchronization of the display of themusic to account for the actual performer's timing, and any errors ordifferences. Automated Mode 6 is analogous to A Mode 1 without override,but not to A Mode 3, which auto-advances independent of the user'sperformance. At step 1153, the automated modes 6 is entered. At step1310, analogous to step 272 in FIG. 2D, the user's performance istracked and stored for comparison to the musical composition displaypresentation relative to the timing of where the performance should be.At step 1320, the performance that has been tracked is compared to thescore and deviation performance data for that performer is generated. Atstep 1330, the errors are utilized to resynchronize the displaypresentation of the musical composition to adjust to the actual user'sreal time performance timing relative to the display presentation and asynchronization of the commencement of the user's performance relativeto the musical composition's presentation on the display. At step 1340,an analysis is done as to whether processing is complete (e.g. is themusical composition done?). If so, A Mode 6 ends at processing step1350, and either the automated modes of processing can end, or otherautomated modes can be checked for, and processing can begin for anotherselected automated mode.

Referring to FIG. 14, there is illustrated the communicationsarchitecture and system in accordance with the present invention,wherein there are a plurality of individual music workstations 1410,each responsive to a user's performance of a musical input (1411) whichis processed by the individual music workstation 1410 to provide aperformance data output 1412 coupled for communication to a controlsystem 1450, which can be a master controller or master workstation,which is coupled for receiving and processing the individual workstationperformance data for the plurality of individual workstations. Acommunications interface 1413, such as a modem, can provide for couplingand communication between the individual workstation 1410 and the mastercontroller 1450. In one embodiment, the modems 1413 are coupled via apublic switched telephone network to a corresponding modem 1455 whichcouples to the master workstation to communicate by directionallybetween the individual workstations and the master workstation.Alternatively, any sort of cable, hardwired, or wireless communicationsinterface and provided, and would work equally well when in accordancewith the present invention. The master controller 1450 is responsive tothe individual performance data outputs from the plurality of individualworkstations to provide a combined and synchronized virtual performancedata output 1451 which is coupled via the communications network to theplurality of individual workstations, which each then respectivelyprovide an output (e.g. audio) 1415 for presentation to the user,representative of the combined performance data. An audio and/or visualpresentation output 1452 can also be provided at the master workstation.

Referring to FIG. 15, there is illustrated the data flow andcommunication architecture for one embodiment of the present invention.A music communication system is provided that has a plurality ofphysically separate remotely located locations, each having one or moreindividual music workstations, wherein each of the individual musicworkstations provides for communication output of individual performancedata for a user's performance of a displayed musical composition at thatindividual workstation. As illustrated, a master controller 1505 iscoupled to the communications infrastructure 1510 at a first location. Aplurality of individual music workstations 1511, 1512, 1513, and 1514are coupled to the communications infrastructure 1510 at location 1,designated performance group 1. Similarly, location 2, performance group2, provides individual workstations 1521 and 1522 coupled to thecommunications infrastructure 1510, and location 3, performance 3,provides individual music workstation 1531, 1532, and 1533 coupled tothe communications infrastructure 1510. The plurality of individualworkstations 1511, 1514, 1421-1522 and 1531-1533, each provideindividual performance data, providing multiple discrete time samples ofperformance data (e.g. time, segments of data), which can includesynchronization information therein, which are communicated via thecommunication infrastructure 1510 to the controller or masterworkstation 1505 The resultant output (e.g. audio) from the individualworkstations is the resynchronized combined performance data of theplurality of individual performance data, resynchronized for each of thetime segments of output.

Referring to FIG. 16, the data word structure for one embodiment of thepresent invention is illustrated. The data structure for the individualuser performance data is illustrated as comprising a header comprised ofcontrol information including a start of word, cyclic redundancy code(CRC) word, the performer's ID (a unique address for the individualworkstation, (which not only can provide an unique ID but can alsoprovide for indication of an instrument type), and other controlinformation for other modes of operation in addition to the virtualperformance), payload data representing the actual performance data, andtiming synchronization data. Any other data word structure providing forheader information including control information, and payloadinformation including performance data, and some timing synchronizationscheme, whether or not part of the user performance data word will workequally well with the present invention.

FIG. 17 illustrates a networked system embodiment in block diagram form,illustrating the network interface subsystem and music data processorsubsystems for one embodiment of the present invention, compatible withthe architecture as described in FIGS. 1-15. FIG. 17 illustrates asingle individual workstation 1710, although a plurality of individualworkstations 1710 are usually involved in the networked system(analogous to the plurality of individual workstations 1410 of FIG. 14).The individual workstation 1710 provides for communication with a masterworkstation 1790, (analogous to the communication architecture with themaster workstation 1450 of FIG. 14). The musical performance by the user(1712) is coupled to an input interface 1715 which provides musicalperformance data to a musical data processor 1720, which can be a MIDIbased processor, which provides the music processed performance dataoutput to a network communications interface 1730 which outputs theindividual musical performance data 1731 for coupling via thecommunications infrastructure to the master workstation 1790.

The master workstation 1790 processes the individual musical performancedata 1731, from the plurality of individual workstations 1710, andprovides an output 1791 of combined virtual performance data, whereinboth the individual musical performance data and the combined virtualperformance data are comprised of a plurality of discrete time segmenteddata samples, wherein the combined virtual performance data issynchronized within each sample time to provide for synchronizedcombination of the individual musical performance for that time segment

A network interface subsystem 1735 is responsive to the combined virtualperformance data to provide synchronized virtual performance data outputto the music data processor 1725 (e.g. MIDI) which provides a soundoutput of the combined virtual performance data corresponding to thesynchronized combination of the individual musical performance data froma plurality of the individual workstations 1710. Additionally, theindividual workstation 1710 provides data analysis and presentationprocessing 1740 which provides an output responsive to, inter alia, thecombined virtual performance data to a presentation means such as anaudiovisual display, or a video display or an audio output, 1745. Notethat in a preferred embodiment, the multiple individual networkworkstations 1710 are synchronized (e.g. responsive to synchronizationsignal from the master workstation 1790), so that the plurality of theindividual workstations simultaneously commence display presentation ofthe musical composition for synchronization of individual userperformances. In one embodiment, the output of the individualperformance data (such as via a MIDI music processor) is comprised oftiming synchronization information that is used by the masterworkstation to synchronize all of the musical performancesrepresentative individual musical performance data to virtually performtogether in a combined and time-synchronized performance. The combinedvirtual performance data representing the synchronized virtualperformance is then sent back (e.g. broadcast) to the individualworkstations, coupled via the network communication interface 1735 backto the individual workstation's music data processor 1725, to provideaudio and/or visual output corresponding thereto. Where the masterworkstation is for a conductor or teacher, the conductor mode or teachermode can also be utilized in addition to the network virtual performancemod. In fact, all of the automated modes can be selectively combined,such as by providing layering of functions via software.

Referring to FIG. 18, the timing diagrams for the individual performancedata and synchronized virtual performance data is illustrated, relativeto signals output from the individual networked workstations NWS₁, NWS₂,NWS_(N) and a master workstation MWS. As illustrated, the masterworkstation MWS provides a synch signal to commence synchronization ofall workstations (NWS) coupled to the system. In a preferred embodiment,the individual workstations (NWS) begin the display presentation of themusical composition (as does the master workstation) in synchronizationwith the synch signal from the master workstation. Responsive thereto,each of the individual workstations displays its musical composition,and the users of the workstations can perform the musical composition.Individual workstations (NWS) provide individual performance dataresponsive to the user's performance. As illustrated, workstation NWS₁provides performance data A and B at respective times, while in theindividual workstation NWS₂ provides individual performance data C & Dat respective times, and independent workstation NWS_(N) providesindividual performance data signals E & F at respective times. Duringthe first time segment, individual performance data A, C & E are output,respectively, from individual workstations 1, 2, and N. However, asshown in FIG. 18, the relative timing of output of the individualperformance data (and receipt by the master workstation thereof) areskewed from one another in timing. The master workstation resynchronizesthe individual performance data A, C, and E within each time segment,and provides an output of the recombined reconstructed and synchronizedA, C, and E to provide synchronized combined virtual performance data.It is to be noted that synchronized virtual performance data A, C, Eoccurs within the first time segment T1, which contains both theindividual performance data and the corresponding responsivesynchronized virtual performance data. The maximum time delay T_(D)represents the acceptable worst case time that a user can toleratebetween the user's performance output of corresponding individualperformance data and the time until the synchronized virtual performancedata corresponding to that individual performance data is received bythe same individual workstation and provided as audio output. Similarly,during time slot 2, each of the individual workstations providesindividual performance data B, D and F, slightly time skewed from eachother, and the master workstation provides combined synchronized virtualperformance data B, D, F, which is communicated back to the individualworkstations to generate an audio output.

The present invention finds application in many different environmentsand embodiments. For example, a live practice can be held from multiplelocations, while one or more musicians are editing the score andcommunicating the changes real time, reperforming them, etc. Anorchestra can be at one location, while a singer can be at anotherlocation. Creating and editing and synchronizing music to movies scorescan also be facilitated with the present invention with beneficialgains. For example, in a movie, a lot of elements are changingdynamically that need to be resynchronized. For example, a scene can belengthened or shortened, and the soundtrack must be adjusted. A scorewhich was created to the movie must be rewritten to accommodate theedited movie. Typically, the composer creates the score at home or at astudio. That score is sent to the movie studio. However, the moviestudio may edit the scene that that music is used in, say a reduction ofthree seconds in the scene. The edited scene is sent back to thecomposer so that he can shorten his music. The process is slow,inefficient, and an impediment to the production process. By utilizingthe present invention, this can be accommodated in real time, and fromremote locations, and the virtual performance mode as well as the othermodes of the music communication of the present system can be utilizedadvantageously for this application.

The present system provides an infrastructure for a music Internet. Itcan be applied to interactive editing of music. It can provide forspecial one on one mode for teaching and performance evaluation. It canfacilitate adapting a music display to the performer's performance, ordrive the display to a metronome and force the user's performance to besynched to the metronome timing. Music can be composed, edited,transposed, etc. easily, from any of the plurality of individualworkstations or on a single standalone workstation. Multiple individualmusic workstations (INW's) can communicate with one another, both withdata and audio such as voice and/or music, and/or supplemental data suchas video, audio, data files, control signals, applets, etc. Bands canpractice at home without the disadvantage of either practicing alone orhaving to physically get together in a large grouping Singers can groupwith other instruments and other vocalists via respective ones of aplurality of IMW's, one, two, or even the whole plurality being remotelylocated physically from one another (e.g., LA, NY).

There can be multiple instruments playing at a single location. Thereare many options regarding how IMW's are allocatable, such as. (1) onIMW (Individual Music Workstation) per location, (2) IMW per instrumenttype, (3) one IMW per performer (e.g. Instrument, vocal). There are alsomany options (1) for networked communications between the IMW's and/orbetween the IMW's and the Master Workstation. Modes can be selectivelymixed and matched and combined, such as (1) all auto-modes enabled mode(2) virtual performance mode, full audio-visual, (3) audio link onlymode, etc. There are also many options as to modes of signal processingwhich can be applied to the user input to generate the individualperformance data, ranging from simple analog to digital (A/D), tocomplex musical sound analysis algorithms (e.g. to detectcharacteristics such as instrument type, pitch tone, attack, decay,amplitude, voice signatures, etc.) There are also tremendousnumbers/parameters of performance analysis algorithms that can beselected and which utilize the selected musical composition data(corresponding to that displayed on the IMW which is receiving theperformance input for analysis) (e.g. the sound characteristicsgenerated by the sound analysis of the user's performance, to (3) inbetween "(1)" and "(2)". Additionally, many kinds of data can becommunicated between (1) IMW's (2) IMW's and a Master Workstation, and(3) IMW's and external communication interfaces, etc. Examples of thekinds of data include performance data, video data, processed/analyzedperformance data, audio signals, data files, etc.

Referring to FIG. 19, the memory work space structure for both theindividual and the master workstations, in the virtual performance mode,is illustrated. Each of the individual workstations provides for musicdata buffering o the outgoing live performance data, individualperformance data IPDXXX, and further provides for storage of theincoming combined virtual performance data CPDXXX. This buffering isdone within read-write memory within the individual IMW's. Similarly,the data structure for the memory buffer of the master workstation isillustrated, showing separate buffer storage locations for each of theplurality of individual workstations WS₁ to WS_(N), providing storage ofassociated respective individual performance data IPD A-N. In oneembodiment, multiple frame buffers are provided in the IMS, so thatmultiple time segments of IPD and CPD data can be stored in the bufferstructure, and so that the buffer structure can support both inputstorage and output retrieval operations. Other data structures andbuffering operations are also compatible with the present invention. Thestructure of the data word can be changed or adapted within its existingstructure to facilitate the transfer of supplemental data is transferredfrom the IMS or Master Workstation, or between the IMW's, or between theIMS's and the Master Workstation. By utilizing the illustrated or othercompatible data buffer structure, timing and synchronized (with theframe buffered variety) asynchronous communication are facilitated, notonly as between communication workstations, but furthermore, as to theoperations internally of the IMW. With frame buffering, the individualworkstation and master workstation can dump the result output to thebuffer, or receive input individual performance data asynchronously andbuffered, relative to the other operations of the respectiveworkstation, The IMW can facilitate operation in one mode or in multiplemodes concurrently.

From the foregoing, it will be observed that numerous variations andmodifications may be effected without departing from the spirit andscope of the invention. It is to be understood that no limitation withrespect to the specific apparatus illustrated herein is intended orshould be inferred. It is, of course, intended to cover by the appendedclaims all such modifications as fall within the scope of the claims.

What is claimed is:
 1. A music workstation system for use by a user inproviding a performance of a displayed video display presentation of aselected musical composition, said system comprising:storage means forstoring composition data representative of the selected musicalcomposition; display means for displaying the video display presentationresponsive to the composition data; input means, responsive to theperformance by the user, for providing user performance data output;analysis means for comparing the user performance data to the respectiveassociated composition data; and control means for modifying the displayof the composition data responsive to the analysis means.
 2. The systemas in claim 1, wherein the modifying is comprised of resynchronizing thefixed rate to resynchronize the display of the composition data to theperformance by the user responsive to the analysis means.
 3. The systemas in claim 1, wherein the modifying is comprised of displayingperformance variance data responsive to the analysis means.
 4. Thesystem as in claim 1, wherein the workstation system is furthercomprised of means for modifying the sequencing and timing of thedisplay presentation responsive to input from the users.
 5. The systemas in claim 4 wherein the composition data is comprised of timinginformation data and wherein the sequencing and timing of the displaypresentation is controlled responsive to the timing information data. 6.The music workstation system as in claim 1, wherein thecontrol meansprovides for a display presentation of the differences between thedisplayed composition data and the performance data, responsive to theanalysis means.
 7. The system as in claim 1 further comprising:processormeans for generating a video display output responsive to thecomposition data in the storage means for the selected musicalcomposition; wherein the display means provides the video displaypresentation responsive to the video display output; wherein the inputmeans is further comprised of storage for storing user performance dataconcurrent with the corresponding display presentation; and wherein thecontrol means modifies the video display presentation to reflect theresult of the comparing the user performance, data, and the compositiondata.
 8. The system as in claim 1, wherein the user performance data iscomprised of at least one of audible performance data, visualperformance data, and control data.
 9. A music display presentationsystem for use by a user in viewing a selected musical composition, saidsystem comprising:a workstation apparatus, comprising: storage means forstoring musical composition data representative of musical notation anddisplay presentation corresponding thereto; display means for providinga display presentation responsive to the musical composition data; meansfor progressing the display presentation responsive to the musicalcomposition data; mode controller means for managing the operation ofthe workstation in at least one selected automated operational mode toprovide for automated progression of the display presentation of themusical composition; wherein the selected automatic operational mode iscomprised of at least one of marching band mode, conductor mode,orchestra mode and networked virtual performance mode.
 10. The system asin claim 9 further comprising:input means, for providing an output ofuser performance data responsive to performance by the user analysismeans for analyzing at least one of the user performance data, themusical composition data, and control means for providing an output ofindividual workstation data responsive to the analysis means.
 11. Thesystem as in claim 9 wherein the musical composition data is furthercomprised of temporal data,wherein the display presentation is logicallysequenced responsive to the temporal data.
 12. The system as in claim 9,wherein the workstation provides user instrument type data, wherein themode controller manages selection of the transposition responsive to theuser instrument type data responsive to the user instrument type data.13. The system as in claim 12 wherein the user instrument type data isdetermined responsive to at least one of a live performance by the user,a performance by the user and a user input selection.
 14. The system asin claim 12, wherein the user instrument type data is representative ofat least one of voice, musical instrument by type and electronicallygenerated.
 15. The system as in claim 9, wherein the workstation isoperable in a plurality of operational modes concurrently.
 16. Thesystem as in claim 9 wherein the workstation is coupled forcommunication with at least one other one of the workstations to form anetwork of workstations.
 17. The system as in claim 16 wherein one ofthe workstations in the network functions as a master controller andcollects and stores the individual workstation performance data from theplurality of workstations coupled in the network.
 18. The system as inclaim 17 wherein the master controller processes the stored individualworkstation performance data to provide an output of combined storedperformance data to all the workstations.
 19. The system as in claim 18,wherein the plurality of workstations in the network are synchronized toeach commence the display presentation at approximately the same time.20. The system as in claim 19, wherein the individual workstationperformance data for the plurality of workstation are synchronized andcombined to provide the combined performance data.
 21. A virtual musicperformance system for use by a plurality of users in providing aplurality of display presentations of a selected musical composition,said system comprising:a plurality of individual music workstation(IMW), each IMW comprisinga communication interface providing forcommunications of musical composition data of the selected musicalcomposition with the respective IMW; a computing subsystem providingprocessing and memory for locally storing the musical composition dataresponsive to the communications interface; and a music displayapparatus for providing a local visual display presentation of theselected musical composition responsive to the computing subsystem;wherein the system further comprises means for controlling a pluralityof the IMWs to concurrently provide a plurality of local displaypresentations of the selected musical composition.
 22. The system as inclaim 21, further comprising:an input device responsive to a performanceby the user of the displayed musical composition for providing an outputof user performance data.
 23. The system as in claim 22, wherein thecomputing subsystem provides processing means responsive to said userperformance data for providing an output of individual performance dataout.
 24. A musical data communication system enables a presentation of aselected musical composition for use by a plurality of users, saidsystem comprising:a plurality of individual workstations each providingmeans for independently capturing the musical performance of arespective user and communicating individual performance data; controlmeans for synchronizing and combining the individual performance datafrom the individual workstations to provide an output of combinedperformance data; means for communicating said combined performance datato the plurality of individual workstations; wherein at least one of theindividual workstations provide at least one of an audio, video, andaudiovisual output representative of the combined musical performancesfor all of the communicating plurality of individual workstations. 25.The system as in claim 24 wherein the plurality of individualworkstations provide for synchronized display presentation.
 26. Thesystem as in claim 25, wherein each of the individual workstationsprovide for output of individual performance data representative of themusical performance by the user corresponding to the displaypresentation.
 27. The system as in claim 25 wherein the individualperformance data is further comprised of timing synchronization data.28. The system as in claim 26 wherein the control means provides asynchronization signal, wherein the individual workstations provide saidtiming synchronization data responsive to the synchronization signal.29. The system as in claim 24 wherein each of the individualworkstations is further comprised of a network interface subsystem and amusic data processor.
 30. The system as in claim 24, wherein a pluralityof the individual workstations provide at least one of an audio, video,and audiovisual output representative of the combined musicalperformance.
 31. A musical display system comprising:a memory, forselectively storing musical composition data representative of aselected musical composition; means for producing a display presentationresponsive to the respective musical composition data; selection meansfor determining a selected display format; data processing meansresponsive to the selection means for selectively processing the storedmusical composition data responsive to the selected display format toproduce a corresponding particular display presentation formatresponsive to the musical composition data; and at least one displaysubsystem, responsive to the selected music composition data and theselected display format, for displaying a video presentation of themusical composition in the selected display format.
 32. The system as inclaim 31, wherein said processing comprises at least one of transposingand communicating external to the musical display system.
 33. The systemas in claim 29, further comprising:operational selection means fordetermining a selecting operating mode wherein the data processing meansis responsive to the selected operating mode for controlling progressionof the video presentation.
 34. The system as in claim 31, whereinresponsive to the selection of the display format, the data processingmeans provides for additionally displaying at least one of userfingering and user motions.
 35. The system as in claim 31, wherein theselection of the operating mode provides for selection of the visualadvancement of the display presentation as at least one of scrolling ofdisplay music notation, page turn, page flip, and page push.
 36. Thesystem as in claim 1, wherein the local presentation is a video display.37. A communications system comprising:a plurality of individual musicworkstations, each comprising a music display apparatus for providing alocal visual display presentation of a selected musical composition anda music input for selectively providing a performance data output,responsive to a performance by a user of that respective individualmusic subsystem; wherein the system further comprises means, responsiveto the performance data output from each of the plurality of individualmusic workstations, to provide a combined output of composite virtualperformance data.
 38. The communications system as in claim 37, whereinthe plurality of individual music subsystems each provide means forindependently capturing the musical performance of the respective userand generate respective individual performance data;wherein the meansresponsive to the performance data is further comprised of means forsynchronizing and combining the individual performance data from theplurality of individual music subsystems to generate combinedperformance data; and means for communicating said combined performancedata to the plurality of individual music subsystems; wherein at leastone of the individual music subsystems provide a local presentationrepresentative of the combined musical performances for all of thecommunicating plurality of individual music subsystems responsive to thecombined performance data.
 39. The system as in claim 38 wherein theplurality of individual music subsystems provide for synchronizeddisplay presentation of the musical composition.
 40. The system as inclaim 38, wherein each of the individual music subsystems provide foroutput of individual performance data representative of the musicalperformance by the user corresponding to the display presentation. 41.The system as in claim 38, wherein the selected musical composition isperformed in discrete time segments, wherein the means responsive to theperformance data generates a synchronization signal for start ofperformance, and each of the time segments is synchronized relative tothe synchronization signal from the master means responsive to theperformance data.
 42. The system as in claim 37, wherein the compositevirtual performance data is communicated back to the individual musicsubsystems.
 43. The system as in claim 37 wherein the individualworkstations provide at least one of an audio output and a visualpresentation, responsive to the composite virtual performance data. 44.The system as in claim 37 wherein each of the individual workstations isfurther comprised of a network interface subsystem and a music dataprocessor.
 45. The system as in claim 37, wherein the processor isresponsive to the composite virtual performance data to generate apresentation on the local display.
 46. A method for communication with asource of distribution of musical data representative of a musical scorein a computer-usable form to a music workstation for video presentationof musical notation for the respective musical score, the methodcomprising:accepting at least a part of the musical data from thesource; evaluating the at least a part of the musical data to determineif the musical data is to be downloaded to the respective musicworkstation; selectively storing in local storage the musical dataresponsive to the evaluating; and locally displaying a videopresentation of the musical score responsive to the selectively storedmusical data.
 47. The method as in claim 46, further comprisedof:providing a plurality of separate ones of the music workstations,each of which provides for selectively storing and locally displaying.48. A method for music via performance, for integrating simultaneousmusical performances from a plurality of locations of music displayworkstations into a cohesive whole, the method comprising:acceptingperformance data from each of the plurality of music displayworkstations; processing the live performance data into discrete timesamples; communicating the discrete time samples for combination intocombined virtual performance data; communicating the combined virtualperformance data to at least one of the plurality of music displayworkstations; and providing at least one of an audio and a videopresentation responsive to the combined virtual performance data.
 49. Amethod of providing a video display presentation of a selected musicalcomposition and of a user's performance, said method comprising:storingcomposition data representative of the selected musical composition;generating a video display output responsive to the composition data forthe selected musical composition; displaying the video displaypresentation responsive to the video display output; storing userperformance data concurrent with the corresponding display presentationresponsive to the performance by the user; comparing the userperformance data to the respective associated composition data; andmodifying the display presentation to reflect the result of thecomparing concurrent with the performance of the composition data.
 50. Amusic performance system for use by a user in providing a performance ofa display presentation of a selected musical composition, said systemcomprising:an individual music workstation (IMW) comprising acommunication interface providing for communications of musicalcomposition data corresponding to the selected musical composition; acomputing subsystem providing processing and memory for locally storingthe musical composition data responsive to the communication interface;and a music display apparatus for providing a visual displaypresentation of the selected musical composition responsive to thecomputing subsystem and the musical composition data.
 51. The system asin claim 50, further comprising:association means for associating anInstrument Type to the IMW; control means for broadcasting musicaldisplay data for multiple separate graphical display presentationscorresponding to multiple separate respective multiple Instrument Types;wherein the IMW is further comprised of discrimination means fordiscriminating between the multiple separate graphical presentations toselect a specific one representative of the corresponding respective oneof the Instrument Types, responsive to the association means and thediscrimination means.
 52. The system as in claim 51, furthercomprising:a source of secondary video data representative of asecondary video image; video controller means for displaying thesecondary video image as a picture-in-picture within a subpart of thevisual display presentation.
 53. A communications system comprising:anindividual music subsystem comprising a music input for selectivelyproviding a performance data output, responsive to a performance by auser of that respective individual music subsystem, and a music datareceiver for coupling communicated music data for storage in memory ofthe respective individual music subsystem; and a display for providing avisual presentation to the user from at least one of the stored musicdata and the performance data output from the individual musicsubsystem.
 54. The system as in claim 53, further comprising a pluralityof individual music subsystems, the system further comprising:means toprovide a combined output of composite virtual performance dataresponsive to communicated performance data out from at least two fromthe plurality of the individual music subsystems.